Emotion classification on eye-tracking and electroencephalograph fused signals employing deep gradient neural networks

Emotion produces complex neural processes and physiological changes under appropriate event stimulation. Physiological signals have the advantage of better reflecting a person’s actual emotional state than facial expressions or voice signals. An electroencephalogram (EEG) is signal obtained by collecting, amplifying, recording human brain’s weak bioelectric on scalp. The eye-tracking (E.T.) records potential difference between retina cornea generated eye movement muscle. Furthermore, different modalities will contain various information representations emotions. Finding this modal great help to get higher recognition accuracy. E.T. EEG are synchronized fused in research, an effective deep learning (DL) method was used combine modalities. This article proposes technique based fusion model Gaussian mixed (GMM) with Butterworth Chebyshev filter. Features extraction subsequently calculated. Secondly, self-similarity (SSIM), energy (E), complexity (C), high order crossing (HOC), power spectral density (PSD) for EGG, electrooculography estimation ((EOG-PDE), center gravity frequency (CGF), variance (F.V.), root mean square (RMSF) selected hereafter; max–min applied vector normalization. Finally, gradient network (DGNN) multimodal classification proposed. proposed predicted emotions eight stimuli experiment 88.10% For evaluation indices accuracy (Ac), precision (Pr), recall (Re), F-measurement (Fm), precision–recall (P.R.) curve, true-positive rate (TPR) receiver operating characteristic curve (ROC), area (AUC), true-accept (TAR), interaction union (IoU), also performs efficiency compared several typical networks including artificial (ANN), SqueezeNet, GoogleNet, ResNet-50, DarkNet-53, ResNet-18, Inception-ResNet, Inception-v3, ResNet-101.